Inferring and Executing Programs for Visual Reasoning Supplementary Material

In all experiments our program generator is an LSTM sequence-to-sequence model [9]. It comprises two learned recurrent neural networks: the encoder receives the naturallanguage question as a sequence of words, and summarizes the question as a fixed-length vector; the decoder receives this fixed-length vector as input and produces the predicted program as a sequence of functions. The encoder and decoder do not share weights. The encoder converts the discrete words of the input question to vectors of dimension 300 using a learned word embedding layer; the resulting sequence of vectors is then processed with a two-layer LSTM using 256 hidden units per layer. The hidden state of the second LSTM layer at the final timestep is used as the input to the decoder network. At each timestep the decoder network receives both the function from the previous timestep (or a special token at the first timestep) and the output from the encoder network. The function is converted to a 300-dimensional vector with a learned embedding layer and concatenated with the decoder output; the resulting sequence of vectors is processed by a two-layer LSTM with 256 hidden units per layer. At each timestep the hidden state of the second LSTM layer is used to compute a distribution over all possible functions using a linear projection. During supervised training of the program generator, we use Adam [7] with a learning rate of 5 × 10−4 and a batch size of 64; we train for a maximum of 32,000 iterations, employing early stopping based on validation set accuracy.